Skip to content

Open any Zarr store, not only FLUXCOM NEE - #170

Open
aivanchenk wants to merge 32 commits into
mainfrom
feat/any-zarr-store
Open

aivanchenk wants to merge 32 commits into
mainfrom
feat/any-zarr-store

Conversation

@aivanchenk

@aivanchenk aivanchenk commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #132. Open any Zarr v2/v3 store from the new Data panel (first toolbar button) or a link: ?store=<url>&var=<name>&sel=<dim:index>.

Each store was tested by its own agent, with values checked against an independent read. Screenshots are from the current build unless marked.

FLUXCOM-X-BASE NEE: default, same as main

ACE2S HiRO v2: 6-hourly, julian, chunked by map, partial load

ARCO-ERA5: too large for one load, offers the first days (no CORS, via proxy, round 3)

ESDC 8-daily: 8-day steps, °C

CESM2 monthly: noleap, 0 to 360 longitude

HadGEM3 daily: 360_day calendar

GFDL-CM4 3-hourly

CESM2 piControl: years 1 to 1200

MUR SST: ocean only, packed int16

NOAA GFS: Zarr v3 sharded, hourly

NASA POWER monthly

gridMET: regional grid (no CORS, via proxy, round 3)

CanESM5: Gaussian grid, click near the pole

Unsupported grid (IPSL tos) and no CORS (WeatherBench 2)

Local test stores: no consolidated metadata, time gaps (round 3), integer data types

A Data panel takes a Zarr v2 or v3 URL, with presets, a variable picker and
pickers for extra dimensions such as ensemble member or pressure level. The
open store is mirrored in the address bar so a link reopens it.

The reader finds the time, latitude and longitude dimensions in any order,
decodes CF time units and calendars, applies scale factor, offset and fill
values, and handles 0 to 360 longitudes and regional grids. Every store is
resampled onto days by time-of-day slots so the plots and exports keep
working. Stores chunked by map are capped per load with a partial-load option.

Work in progress: round 1 testing found calendar, axis and copy bugs that
later commits fix.
Values printed with two fixed decimals, so rainfall in kg m-2 s-1 read as
0.00 everywhere. They now keep about three significant digits, with
exponents for very small or large values, while the NEE range keeps its two
decimals.

The line and strip axes always reached down to zero, which flattened
pressure or temperature in kelvin into a line along the top. They now fit
the data, and keep zero only when the data crosses it or sits close to it.

Tiny negative noise in one-sided data such as rainfall no longer spreads
over half the colour ramp; it is clamped to the neutral centre.
An 8-daily or weekly value filled days from its timestamp forward, ran on
into the next year and past the end of the record, and added an empty year
to the picker. Such steps are now taken as centred on their timestamp, stop
halfway to their neighbours, and never cross into another year.

A 360_day year was mapped date by date, so February 29th and 30th rolled
into March and overwrote it, and seven 31sts a year stayed empty. It is now
stretched over the real year by day of year: no value is lost and the spare
days are spread out.

Captions counted filled days as steps ("365 monthly steps"). They now count
the values the store holds for the selected years, with singular forms.
A time axis split into a thousand tiny chunks, as in MUR SST, started every
request at once; the browser ran out of connections and the store was
reported as having no time coordinate. Coordinates now load eight chunks at
a time, and a real read failure is reported instead of being taken for a
missing coordinate.

Packed values outside valid_min, valid_max or valid_range are now missing,
as CF defines, instead of plotting as spikes.
One-sided data keeps a tiny placeholder extent on its empty half so the
scale never divides by zero. Legends printed it as -1e-10; they now read 0.
With one value a day or fewer there is no daily cycle, so the fingerprint
was a single row of stripes. Such stores now get the usual year by season
heatmap: one row per selected year, one column per day, week or month of
the year, with the strips showing the mean of each year and the mean of
each position in the year across years.

The yearly scale and the exported daily-cycle table keep the old layout.
Line resolutions and fingerprint scales finer than the store's own step are
no longer offered, the finest option carries the step's name ("6-hourly"),
and a monthly store opens on its monthly view. The Local/UTC switch shows
only for hourly or finer data: shifting a 6-hourly store by whole slots
labelled Tokyo UTC+12.

Summary cards and the per-year table count and date months for monthly
stores ("Lowest month 2014-01") instead of the days a month fills. Export
copy and the CSV header name the real step, the cell caption keeps
title-case names and does not repeat the step, and the kept-in-memory box
size is capped at the store's chunk size.
A store that never answered locked the Data panel until reload. Opening now
gives up after 30 seconds, and Open and the examples stay usable while a
store is loading, so another one can replace it.

The address field follows the store last asked for, so a failed or slow
?store= link shows its URL instead of an empty field. A mistyped variable
keeps the variable field instead of hiding it. Errors are styled as errors,
say which store is still showing, and read plainly for refused access
(401, 403, 409), answers that are not Zarr metadata, and grids that are not
regular latitude and longitude.

A click outside a regional store no longer snaps to its edge cell: the
title keeps the clicked point and no cell is drawn. The outermost rows of a
global grid now reach the poles, so polar clicks on a Gaussian grid load.
Date.UTC reads years 0 to 99 as 1900 to 1999, so a model run starting in
year 1 opened as a single year 1901 with no data. Every place that builds a
date from a year now goes through one helper that does not.

The year picker shows full years unless a full grid shares one century, so
a single year no longer reads "20", and a long record scrolls to the
selected year. Stores without a title attribute are named from their CMIP
facets, or from the address without its version tag.
Counting downloads alone let a store with whole-globe chunks offer a
partial load of tens of gigabytes within 400 requests (ERA5 temperature on
pressure levels: about 34 GB). A load now also stops at 2 GB of decoded
data, the refusal states how much the full request would take, and the
partial-load button says how much it will download.
The corner map floated over the plot's right edge and the fourth summary
card as the data view scrolled. Large screens now reserve its width as a
right gutter whenever the page margin is narrower than the map.

Also count FLUXCOM-X steps per hour again: its hour of day is its own axis,
so each time sample holds 24 values.
Fingerprint labels ran into each other in the compact panel ("Jan '2Mar
'21") and on long records; a label that would touch the previous one is now
skipped. Records spanning many years label only round years (every 2, 5,
10, 25, ... years) on both plots instead of irregular ones.

A store that cannot list its arrays keeps its variable as a text field
after opening, so it can be switched or corrected. Stores are titled from a
name or dataset_id attribute when they carry no title.
Tooltips compared two points as a percentage, which means nothing for
kelvin, degrees Celsius or a signed flux and blows up near zero. They now
give the difference in the variable's own units. A daily point with a
single value no longer reports "Std 0 across 1 step".

The exported mean daily cycle keyed its rows by slot index, so a 3-hourly
peak at 09:00 read as hour 3. Rows are now keyed by the hour each slot
starts, and stores with one value a day leave the table out.
The focus zoom suited 0.05° cells; a 2.8° model cell then filled far more
than the screen and no outline was visible. A coarse cell is now framed at
the zoom where it spans about 120 px.
…t year

A daily record with most days missing has a typical step of two days, so
each value was spread over two days and the series gained days it never
had. Steps shorter than five days are now read as daily data with gaps;
only weekly and coarser steps are treated as windows.

A store opened on the app's starting year (2021) even when its record ran
to 2026. It now opens on its own last year, while years the reader picked
still carry over when switching stores.
A regional store now opens with the map framed on its area, and the Data
panel lists its coverage; units read "not given in the store" when absent.

Stores often allocate their time axis years ahead (ERA5 to 2050, MERRA-2
to 2029), so the last-year default landed on an empty year. It now stops
at the present year, and an untouched default that comes back empty for
the chosen variable steps back up to two years.

The address field kept snapping back to the previous store after a failed
open; it now shows each new request's address and only the resolved root
of a store that actually opened.
…L field

Exports and the Data panel named the default store "NEE" at
0.049999999999999996°, where they read FLUXCOM-X-BASE at 0.05° before the
refactor. The resolution is now rounded, and the default store carries its
product name and "net ecosystem exchange" again. Exported cell coordinates
drop float noise too.

A time coordinate split into more than 64 chunks is read from its first
chunk and its last value when those show an even spacing, so MUR SST opens
in seconds instead of timing out.

A variable typed for one store is no longer sent with a different address,
typing in the address field is not overwritten by a store that finishes
opening late, and a failed first open says so in the chart panel instead of
describing FLUXCOM. Coverage for stores spanning all longitudes reads
"all longitudes", and such stores are not framed on open.
The line joined the values on either side of a gap, so missing days or a
masked period looked like a straight change. Each run of empty bins (or
empty slots at hourly resolution) now carries one empty point, which the
line does not draw across.
Stores like ERA5 (273 arrays) or MERRA-2 (about 470) listed every variable
in a plain select with no way to search, and long group paths were cut off
at the end. The variable is now a search field that matches the name, the
long name or the group path, and each option shows the array name first
with its groups and long name wrapping beneath.
The variable field for a store that does not list its arrays came up
holding the previous store's variable. It now follows each request.

Year buttons drop to two digits only for a short run within one century
that does not start on a century year, so a 2000 to 2026 store no longer
shows 00 to 26.
Every open logged a handful of 404s: zarrita, left to guess, tried the v2
files (.zmetadata, .zgroup, .zarray, .zattrs) on v3 stores and an array
at the root of v2 ones. The store's format is now read once from its root
and every node is opened in that format, and a root that consolidated
metadata already lists as a group is not probed as an array.
The line caption counted empty days as daily means ("720 daily steps ·
731 daily means" for a 360-day calendar); it now counts bins with data.
Labels under the left strip of the fingerprint skip one that would touch
its neighbour instead of running together, and the exported local hour
keeps its half hour for sub-hourly stores instead of truncating to the
hour.
Stores other than FLUXCOM-X were told an empty cell might be over ocean,
which is backwards for an ocean product like MUR SST. The message now
names a land or ocean mask, or years outside the record. FLUXCOM-X is
recognised by its address, so its vegetated-land wording no longer
depends on the variable being called NEE.
After loading the first 100 days of a store chunked by map, the caption
still gave the whole year's step count (1,453 for ACE). It now counts the
steps the partial load covered.
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

🚀 Vercel preview deployed

Preview URL: https://earth-prints-dzbuztv2w-anastasiia-s-projects10.vercel.app

@aivanchenk
aivanchenk requested review from lazarusA and a balanced review from Copilot November 2, 2026 22:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

aivanchenk added a commit that referenced this pull request Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add an abstraction layer to input any zarr store

2 participants